Multimodal Self-supervised Learning for Medical Image Analysis

Self-supervised learning approaches leverage unlabeled samples to acquire generic knowledge about different concepts, hence allowing for annotation-efficient downstream task learning. In this paper, we propose a novel self-supervised method that leverages multiple imaging modalities. We introduce the multimodal puzzle task, which facilitates representation from image The learned modality-agnostic representations are obtained by confusing modalities at data-level. Together with Sinkhorn operator, formulate solving optimization as permutation matrix inference instead of classification, they allow efficient puzzles varying levels complexity. addition, also utilize generation techniques data augmentation used pretraining, tasks directly. This aims circumvent quality issues associated synthetic images, while improving data-efficiency and methods. Our experimental results show our yields better semantic representations, compared treating each modality independently. highlight benefits exploiting images pretraining. showcase approach on three segmentation tasks, outperform many solutions competitive state-of-the-art.